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ABSTRACT 



This paper reports on the design and results of a paired 
sample survey of 226 graduates and 110 of their employers in one professional 
area at a university. This study was designed to explore the usefulness of a 
method of data collection for paired surveys and the potential benefits of 
surveying both graduates and employers. Survey development and dissemination 
methods are described. A quantitative analysis was made of a list of 22 
general skills. Qualitative analysis of graduate and employer comments 
compared graduate and employer expectations of any graduate employee in terms 
of important skills and their assessments of skills of a particular graduate 
(Achievement of the same skills) . Although few statistically significant 
differences were found between ratings of employers and graduates, results 
indicate that each group can answer different questions that contribute to 
curriculum improvement, and more qualitative methods may reveal differences 
in similar questions. Recommendations for survey development and use include: 
(1) supplementing questions on general skill levels with open-ended 
questions; (2) differentiating between the importance of and achievement of 
skills; (3) graphically presenting qualitative results for decision makers ; 
and (4) using alternate question types and qualitative data collection 
methods to control for respondents' tendencies to provide predominantly 
positive responses. (Contains 2 figures, 4 tables, and 16 references.) (SLD) 
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ABSTRACT 

This paper reports on the design and results of a paired sample survey of 
graduates and their employers in one professional area at a university, to explore 
the utility of a method of data collection for paired surveys, and potential 
benefits of surveying both graduates and employers. Survey development and 
dissemination methods are described. Results present quantitative analysis of a 
list of 22 general skills as well as qualitative analysis of written comments to 
compare employers’ and graduates’: a) expectations of any graduate employee 
(important skills), and b) assessment of skills of a particular graduate 
(achievement of the same skills). Discussion includes a description of how 
survey results were used for curriculum revision and recommendations for 
future improvements. Although few statistically significant differences were 
found between ratings of employers and graduates, the authors recommend 
surveying both populations. Each group can answer different questions that 
contribute to curriculum improvement, and more qualitative methods may reveal 
differences in similar questions. Recommendations include: a) supplementing 
questions on predetermined general skill areas with open-ended questions; b) 
differentiating between the importance and achievement of skills; c) graphically 
presenting quantitative results for decision makers; and d) using alternate 
question types and qualitative data collection methods to control for 
respondents’ tendency to provide predominantly positive responses. 
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Background 



Continuing trends of accountability, quality assurance, and the incorporation of a 
customer focus in higher education will increase the necessity for educational 
institutions of all types to collect data from their “end users.” The data collected from 
students after graduation or their employers can be a valuable tool in the evaluation of 
outcomes and planning of curricula. Programs can be revised to accommodate changes 
indicated by these reviews. Although surveys can require a great deal of money and 
staff time, they still represent the most cost and time efficient method of collecting data 
from a large number of people. They therefore are a common, and sometimes the only, 
method used to collect information for purposes of assessment and revision. 

This paper reports on the design and results of a paired sample survey of graduates and 
their employers in one professional area at a university, to explore: 1) the utility of a 
method of data collection for paired surveys; and 2) potential benefits of surveying both 
graduates and employers. The surveys were conducted in 1996 by the Department of 
Building and Real Estate (BRE) in the Hong Kong Polytechnic University. Hong 
Kong Polytechnic University is the largest of the seven universities in Hong Kong with 
over 1,000 full-time faculty and over 20,000 students (13,000 full-time equivalent). 

The Department of Building and Real Estate has 38 full-time faculty and admits more 
than 270 new students every year. The department’s four undergraduate degree 
programs are accredited by professional organizations in the United Kingdom. The 
courses are very vocational in focus, with graduates going on to complete 2 years of 
professional practice before assessment and certification of their competence by these 
professional organizations. 

There has always been a need for building surveying and construction degree programs 
to closely follow any changes in the skills required by the property and real estate 
market and to reflect these in courses offered. A lack of responsiveness in such courses 
has led, in the UK, to an identified lack or mismatch of skills offered by Surveying 
degree programs in the areas of linguistic and management skills and financial analysis 
(College of Estate Management, 1992). 1 The Department of Building and Real Estate 
has regularly sought informal advice from employers, and consults with an advisory 
panel whose members are recognized leaders in the local industry. BRE decided to 
collect more systematic feedback from employers and graduates as part of a larger 
curriculum revision effort. 2 This data was to be collected through self-administered 
surveys. 

A review of the literature on employer and graduate surveys revealed that most of the 
work was unpublished, and that there was considerable variability in the focus and 
quality of such surveys. Some surveys had attempted to identify the skills, knowledge 
base and educational goals considered desirable by employers (Moohan [1993] and 
Otter [1992] are examples), while other surveys had elected to track students after 
graduation to gain feedback on their perceptions of college and its value in 
accomplishing transfer and employment goals (Frank, 1991; Pettit, 1991; Kiger, 1994; 



1 The surveying profession in the UK comprises 7 divisions; the main three covering “Quantity 
Surveying”, “ General Practice” or “Real Estate”, and “Building Surveying.” The professional expertise 
of “Surveying” is not to be confused with the data collection method of “surveying” reported in this 
paper. 

2 The university is phasing in a credit-based system which will begin with first-year students in the fall 
of 1998. As in the present system, students can only begin university if they have been accepted into a 
particular three-year course of study in a department. However, with the modularized credit-based 
system (drawn from American examples) each year cohort will no longer progress through the course 
on the same schedule and students will be able to select electives and take subjects from outside their 
major department. This change to the credit-based system provided an opportunity to examine existing 
courses and implement substantial curricular changes. 
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Brennan et. al., 1993). Reported studies rarely collected data from both graduates and 
employers. In the few examples found, only one study used matched graduate and 
employer surveys (Banta, 1993). 

It quickly became apparent that linked graduate and employer surveys are 
underrepresented in the literature. As expectations increase that institutions will collect 
information from employers and/or graduates, there certainly is scope for further 
research and methodological development. Therefore, in the present study it was 
decided to collect data from both graduates and employers, not only for the customary 
uses of assessing outcomes and informing curriculum revision, but also to advance 
survey research in this context. As surveys of this type are conducted more often, it 
will be important to arrive at cost effective means of collecting the most valid data 
possible. This study explores whether there are benefits in consulting both stakeholder 
groups, and begins to answer questions such as: 

Do employers and graduates agree on the most important skills for the 
workplace? 

Do the abilities of the graduates match employers’ expectations? 

Do graduates feel they were well-equipped for the workplace by then- 
education? 

Do employers agree with the graduates’ assessment? 



Study Design and Methods 



Questionnaire development and dissemination 

The questionnaire surveys for employers and graduates went through several stages of 
development. In the spring of 1996 a series of department-wide faculty and special 
small group meetings resulted in a decision to survey the opinions of both recent 
graduates and their employers, and the general areas of inquiry of the surveys. Surveys 
were then developed by a task force comprised of subject matter experts within the 
department, and external educational consultants. First drafts of the surveys were 
based on information from the literature search and input from department faculty. The 
surveys were then repeatedly revised to accommodate suggestions from consultations 
with employers (through an advisory panel), and resolve potential problems identified 
in two cycles of piloting on target respondents. 

In the final version, the questionnaires collected information in four areas: 1) general 
information on the respondent, 2) assessment of importance of skills for all graduates, 
3) assessment of a particular graduate’s performance of these same skills, and 4) open- 
ended questions requesting written comments (see Table I for a detailed description of 
each section). The final survey package consisted of three and four page surveys for 
employers and graduates respectively, a one page explanatory cover letter, and either a 
stamped addressed envelope for return mailing (graduates) or return fax instructions 
(employers). 



Insert Table I about here 



Except for minor changes to maintain clarity, the order, wording and scales for the 
twenty-two closed response items used in the second and third sections of the surveys 
were kept the same to facilitate comparing responses from the two groups. The twenty- 
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two items included general skill areas that both department staff and employer 
representatives had agreed covered the basic skills that graduates of any program in the 
department should be assessed for. The list was generated through brainstorming 
sessions and refined over time through voting and discussion processes. It resembles 
the skill areas used in numerous other surveys found in our own literature review as 
well as the recurrent themes sought by employers across a wide range of fields in the 
studies reviewed by VanHom (1995). In addition, several of the open-ended questions 
were kept as similar as possible so that comparisons could later be made between 
employers’ and graduates’ written responses. 

Methods for survey administration (as well as development) were drawn from Bourque 
& Fielder (1995) and Dillman (1978). To maximize response rate, three waves of data 
collection were carried out and a tracking system was implemented to follow-up with 
non-respondents. Survey dissemination methods were slightly different for graduates 
and employers to take their likely preferences into account. Graduates were first mailed 
the survey with an explanatory cover letter and stamped addressed return envelope. 

Both the cover letter and survey clearly requested they complete and return the 
questionnaire by a specified due date (two weeks from date of mailing). This was 
followed by a letter either thanking them for their response or noting that their survey 
had not yet been returned and asking them to do so. After another two weeks a “second 
edition” of the questionnaire was mailed with a different cover letter and stamped 
addressed envelope to the graduates who had not yet responded. Employers were 
faxed the cover letter and survey and requested to return fax their responses within a 
week. Each employer also received a thank you or reminder fax after the week had 
passed, and non-respondents received a revised cover letter and new copy of the survey 
after another week had passed. 

A tracking system was essential for thanking respondents and following up with non- 
respondents. Each graduate and employer was assigned an identification code, the 
purpose of which was clearly explained in the cover letter. To preserve confidentiality, 
the identification code was cut off of returned questionnaires after the code had been 
noted in a separately kept log on who had responded. Also as a gesture of thanks and 
to improve response rate, respondents were invited to return their name and address on 
a separate form if they wished to receive a copy of the survey results. A larger number 
than expected requested this information (44 of the 110 employers and 97 of the 226 
graduates). 

Employer names were generated from graduates. Graduates who were working in the 
field were asked to supply a name and contact number of a supervisor familiar with 
their work. In contrast to other surveys using this method reported in the literature 
(Banta, 1993), a large percentage of graduates complied (180 of 187 graduates working 
in the field, or more than 96% of qualified respondents). Specifically named employers 
were then directly contacted. Employer survey forms included a space in which the 
name of a particular graduate (who had identified the supervisor as familiar with his/her 
work) was written. This mechanism assured that only someone familiar with the 
graduate as an employee completed the matched employer survey. To preserve 
confidentiality, the graduate’s name was also removed after responses had been coded. 
Data tables for both employers and graduates contained matching information, but only 
based on the identification codes. 

Questionnaire analysis 

The entire population of graduates from 1994 and 1995 were surveyed. These years 
were chosen for the theoretical reason that they would be new enough to the work 
environment to be able to both recollect what they had learned in school and comment 
on other skills new graduates most needed, and for the practical reason that the database 
would contain the most accurate addresses for these more recent graduates. A total of 
442 questionnaires were sent out to graduates. 226 were returned for a 51% response 
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rate. Of the returned questionnaires, 187 indicated they were employed in the field. 
Surveys were sent out to all 180 employers named by graduates. 1 1 0 employers 
returned surveys for a 61% response rate. 

Analysis of the surveys included quantitative and qualitative methods. Closed ended 
questions were coded into numerical responses and entered into the statistical program 
SPSS from which descriptive and explanatory statistics could be run. 1 14 paired 
employer and graduate surveys comprised the data set for quantitative analysis. 3 Open- 
ended questions were analyzed qualitatively using a system of coding and categorizing 
all written responses, and testing for coder and category reliability as described in Fink 
(1995), Lincoln & Guba (1985), and Tesch (1990). 



Results 



The surveys provided a large amount of data that was analyzed in different ways and 
put to a variety of uses for the purposes of outcomes assessment and curriculum 
revision. As the purpose of this paper is to describe a method of matching graduate and 
employer surveys and explore the utility of the results, only a few sections of the data 
will be focused upon. More specifically, we will center on the 22 closed response 
items of the paired sample to compare employers’ and graduates’: a) expectations of any 
graduate employee (important skills), and b) assessment of skills of a particular 
graduate (achievement of the same skills). We will also briefly report on one open- 
ended question collecting written comments from both employers and graduates on 
essential skills not included in the list of 22. 

Do employers’ and graduates’ views differ on what skills are important in the 
workplace? 

This question was explored through two processes. Paired samples t-tests (two tailed) 
were run on the 22 items. Written comments of the two groups were also qualitatively 
analyzed and compared. 

The results of the paired samples t-tests are reported in Table II. At a significance level 
of p < .05, there were no significant differences between employers’ and graduates’ 
opinions of the importance of 16 skills. 4 Of the remaining six skills, “extensive 
practical knowledge,” “oral communication skills,” and “leading others effectively” 
were rated as significantly more important by graduates; and “computer literacy,” 
“working co-operatively as a team member,” and “promoting truth and ethical 
standards” were rated as significantly more important by employers. 

Given anecdotal evidence gathered across years of conversations with both employers 
and graduates, the researchers and department faculty had expected many more 
significant differences. Even what new knowledge was generated from the six 
statistically significant results was questioned. As the target population had a maximum 
of 24 months experience on the job, it was no surprise that they wished for more 
extensive practical knowledge. For the same reason, graduates may feel it is more 
important to know how to lead others, in contrast to their employers’ perception that 
this was not important at such an early career stage. Two other significant results 
brought up a question as to the validity of the items themselves. To what extent could 
“possessing oral communication skills,” significantly more important to graduates, be 
considered a part of “working cooperatively as a team member,” a skill significantly 
more important to employers? However, the fact that these differences cannot be 
accounted for by chance should not be forgotten. The paired design of this study 



3 There are slightly more pairs than employers because in a few instances more than one graduate 
identified the same supervisor. 

4 If a level of p < .01 had been chosen, two skills (1 & 11) would have been significant. 
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controls for many potentially confounding variables. Why these differences exist 
should be explored, especially to discover implications for curriculum revision. 



Insert Table II about here 



Both forms of the survey asked respondents to write in any other skills that were 
essential for graduates. This question immediately followed the 22 item skills list. All 
comments were compiled into two separate lists of employers’ and graduates’ 
comments. 5 These lists were reviewed for themes. Shorthand codes were then created 
for these themes, the lists were repeatedly reviewed and attempts were made to attach 
codes to each of the comments. After several cycles of adjusting themes/categories and 
recoding the individual comments, a limited number of discrete categories emerged 
which accounted for most of the comments on each list. The three categories which 
account for more than half the comments on each list are detailed in Table HI. It was 
interesting to note that for this, as for all other open-ended questions in both surveys, 
the number of comments fitting into a single category dropped off sharply after the three 
most common categories. More important to this particular study, employers and 
graduates identified the same three categories of additional essential skills, and in the 
same order of frequency. More than half of the comments from both groups were 
either general statements about the importance of, or requests for specific a) intellectual 
skills; b) technical skills; or c) practical training. 



Insert Table HI about here 



In summary, the quantitative analysis of the 22 item skills list and qualitative analysis of 
written comments on additional essential skills suggested there was little practical 
difference between employers’ and graduates’ views on the importance of various skills 
in the workplace. As has occurred in many other surveys of this type, there was a 
tendency to rate all the skills as important. The means for all 22 items for both groups 
was above 2.5 on a four point scale. 6 There are several plausible explanations for this 
tendency of respondents to rate all the skills as important. One possibility is that, as 
experts in the field had designed the list to cover what they felt were the important 
skills, it would be no surprise if employers and graduates agreed with them. Another 
explanation is that the response scale was not as clear and appropriate as possible. 
Whatever the reason, respondents’ use of a limited range of the scale may make 
statistical significance less common, and certainly restricts the understandings that can 
be derived to help make practical decisions. For example, the differences between 
means of the 22 items was often as little as a few hundredths of a point. A faculty 
decision to concentrate their improvement efforts on any portion of the skills ranked as 
most important would therefore be arbitrary. 



5 Note that qualitative analysis of written comments included all respondents, not just the 114 pairs. 

6 In fact, most of the means were much higher. The “mean of the means” was 3.27 for graduates and 
3.24 for employers (for those who don’t completely discount this “average of averages”). 
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Do employers and graduates differ in their assessment of achieved skills? 



This question was explored via paired samples t-tests (two tailed) on 22 closed 
response items asking respondents to assess the achievement of a particular graduate in 
each of the items previously rated for importance. The results of the paired samples t- 
tests are reported in Table IV. At a significance level of p < .05, there were no 
significant differences between employers’ and graduates’ opinions of achievement of 
16 skills. 7 Of the remaining six skills, graduates reported significantly higher 
achievement in possessing “extensive practical knowledge,” being able to “define and 
solve problems,” and “work autonomously;” and employers reported graduates 
achieved more in the skill areas of “listening,” “working co-operatively as a team 
member,” and “being creative.” 

Once again, researchers and department faculty had expected many more significant 
differences in these items. As many faculty had heard complaints about the quality of 
graduates from the advisory panel as well as employers they interacted with, the 
expectation was that employers would rate graduates significantly lower in most of the 
skill areas. 8 It came as a pleasant surprise that although the graduates had rated 
themselves higher in three areas, there were also three areas which the employers rated 
graduates higher than they had rated themselves! 

It was interesting to note that only two of the items with a significant difference in 
importance rating also received a significant difference in achievement rating. 

Graduates rated possessing “extensive practical knowledge” higher in importance and 
achievement, and employers did the same for “working cooperatively as a team 
member.” This was useful information as during the survey development stage many 
of those involved wondered whether employers would rate graduate achievement lower 
for more important skills, and conversely, whether graduates might tend to rate as more 
important the skills they felt they were still achieving the least in. 



Insert Table IV about here 



Quantitative analysis of the 22 closed response items indicates much less difference 
between employers’ and graduates’ perception of achievement of various skills than had 
been expected. Possible confounding variables include a tendency for employers to 
provide higher scores when a specific graduate is named (Banta, 1993), or conversely, 
modesty on the part of this particular group of graduates (an often repeated stereotype 
of Asians). A slightly larger portion of the scale was used in answering the 
achievement questions, but the means for 21 of the 22 items for both groups was still 
above 2.5 on the four point scale. 9 Once again, use of a limited range of the response 
scale is troubling. Plausible explanations of this tendency to rate graduates “above 
average” for all skills include: a) graduates are really doing a good job and achieving 



7 If a level of p < .01 had been chosen, three skills (1, 4 & 9) would have been significant. 

8 This concern generated a great deal of discussion during survey development. Many faculty members 
were concerned that disseminating a survey asking questions related to achievement would lay the 
department open to criticism and attack. 

9 The one skill eliciting an achievement mean of just under 2.5, “ability to communicate in 
Putonghua” (the national language of Mainland China) was not even part of the curriculum for 
students! (Over 90% of graduates are native speakers of Cantonese and have received no formal 
language training in Putonghua, though many Cantonese speakers have shown the ability to pick up 
Putonghua when required for business purposes.) 
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that well; b) employers feel graduate achievement is high given relatively low 
expectations of performance as new employees; or c) employers do not wish to appear 
overly critical. However, the “real” explanation (or explanations) is not known, and 
the data does not clearly indicate a desirable course of action for making improvements. 



Discussion 



As is common to all studies, the researchers found some processes and tools beneficial 
and plan to employ them again in suitable situations. Other activities were less 
beneficial and are not likely to be repeated. After briefly reporting how the survey 
results were used, we will describe what we would do again, and what we would 
change, given a similar study. 

As has been described, all members of the department had been involved to varying 
degrees in the development of the surveys in the spring of 1996. At the very least, they 
had attended several meetings in which they had brainstormed the list of skill areas 
essential to all graduates, and then voted to arrive at the final list. The survey was then 
conducted over the summer. Just before the fall semester began, staff participated in a 
two day retreat devoted to revision of the four undergraduate degree curricula. The 
first day of the retreat was used for presentation of survey results and decision making 
based upon those results. Staff were given the full list of written comments from 
employers and graduates and prioritized lists of the themes generated. They also 
received quantitative results in the forms of frequencies, and ranked and plotted means. 
Staff reviewed survey data in small groups to generate prioritized lists of aspects of the 
existing curricula to preserve, and new areas of focus for the revised curricula. 

The greatest benefit of the survey was how it focused and galvanized the energies of the 
department. Involving staff in the development of the survey at the beginning of the 
curriculum revision process focused them on teaching and their learners. Ideas 
contributed by staff substantially improved the final surveys. Processes associated with 
the survey also appear to have lead to greater staff commitment as well as smoothed 
later stages of the curriculum development process. Evidence of this commitment 
includes a substantial portion of staff remaining in the halls to animatedly continue 
discussion of curriculum issues after the close of the two day retreat (after 5 p.m. on the 
Friday before classes were to start)! Almost every staff member later took 
responsibility for drafting at least one new subject in the new curricula. Survey results 
and the prioritized lists of goals generated during the retreat contributed to the first draft 
of these modules being noticeably cohesive. 

In a similar situation we would again ask questions on general rather than discipline 
specific or technical skills. It is beneficial to focus on general skills. Through 
protracted discussions it became apparent that even in a single department there was so 
much variability in the degree courses, let alone the nature of the jobs students entered 
after graduation, that it would be impossible to come up with a single survey that asked 
questions on specific skills for all graduates. Even if different forms of the survey were 
developed for graduates of different degrees, the specific skills are so numerous that a 
survey would be too long and cumbersome and few would be likely to complete it. 

And most importantly, staff representing different disciplines and degree courses in the 
department came to realize that what was most important for their students’ future 
careers as well as the industry, was that they became proficient in general skills. To 
paraphrase one staff member, “Its more important that they know how to look up the 
information than that they recall the dimensions of a brick. The properties of bricks will 
keep changing after our students graduate - what our students need is to know how to 
keep up with this new technical knowledge.” 

However, it is useful to supplement questions on predetermined skill areas and those 
with closed responses with open-ended questions. Comments generated from open- 
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ended questions were relatively simple to compile and analyze for trends, and the 
valuable information gathered from them more than justified the additional effort 
required. A few open-ended questions that are carefully worded for clarity and placed 
immediately after the list of general skills can generate very useful written comments 
from respondents. We would also recommend questions asking respondents to identify 
problems and generate solutions on issues they are qualified to answer (an example is 
asking graduates to list the three strengths of their degree program and three areas most 
in need of improvement). One obvious benefit is that respondents may list important 
skills that are newly emerging or have been missed by survey developers. We also 
found that the written comments of respondents were remembered by staff. Written 
comments added richness of detail and personalized the issues. Information gained 
from these qualitative questions often helped clarify understanding of quantitative 
results - especially important in this situation in which there were relatively few 
significant quantitative results. 

The combination of items on general skill areas and open-ended questions can serve as 
a flexible design for surveys of this type. The surveys reported on in this paper have 
been developed with the intention of expanding their use university-wide. Although 
this “first draft” was developed by a single department, the general skills list would 
apply to most of the programs in the university. There would also be advantages in 
gathering this data across a wider range of employers and graduates. Additional 
department-specific closed response items, combined with open-ended questions, could 
gather discipline specific information. By beginning with a basic survey, each 
department could spend less time in survey development, but still refine and add 
questions as appropriate. 

We also recommend reviewing the literature before each new survey effort as new 
developments could improve flaws in survey design. For example, it was only after 
this survey had been sent out that an article describing stages in new graduates’ 
transition from higher education to work (Sagan, 1990) was discovered. As the 
graduate population surveyed in our study covered several stages hypothesized by 
Sagan, it is possible our study results are confounded by this phenomenon. 

There are also several techniques used in the survey design or reporting of results that 
we would use again. First, we will continue to clearly differentiate between the 
importance and achievement of skills. Questions related to importance collect 
information on the scope and priorities of the industry. New areas may be demanded 
on the job that are not even addressed in the curriculum. Questions related to 
achievement provide feedback on how well graduates are prepared for the workplace. 

In this particular study, respondents appeared to clearly distinguish between importance 
and achievement, but asking both types of questions can be a control for this potential 
confusion in future surveys. Also, it is necessary for decision makers to keep the 
difference between importance and achievement in mind. For example, in the present 
study we found the skill area graduates had the lowest achievement in was also rated as 
one of the least important! 

This leads to another technique we will reuse. Graphing quantitative results provided 
much clearer understanding from which staff could make decisions. In particular, we 
would recommend graphing skills to compare importance and achievement. As 
described earlier, in the present study respondents tended to limit their responses to a 
small portion of the range and the differences between the means were often so small to 
be insignificant for practical use. This meant a ranking of means for the 22 general 
skills was misleading. We found graphing the means with importance on the x scale 
and achievement on the y scale had two benefits. First, it was much easier to see 
relative distances between means. Second, when the graph was divided into four 
quadrants (using the overall means for “importance” and “achievement”) it was much 
easier to perceive areas for improvement as well as those in which to maintain the status 
quo (see Figures I & II). 
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Insert Figures I & II about here 

Given what we learned from conducting these surveys, there are also some things we 
would change. First, the restricted use of the positive end of the response range by 
both graduates and employers is troubling. We would attempt to better control for this 
by augmenting standard closed-response scales with other kinds of questions. Two 
possibilities are ranking or distributing a set number of points across a list of skills. 
Also, as so much was learned from qualitative analysis of written comments, we would 
strongly consider using additional qualitative methods in future surveys. For example, 
following up with a portion of respondents via phone interviews could both collect 
more detailed information and perhaps act as a control for the response bias of more 
quantitative questions. We must also remain aware that items can be differently 
interpreted by respondents. Both literature review prior to survey development and 
incorporation of qualitative data gathering techniques can help prevent the use of, or 
control the damage from, inappropriate items. In the future we may also experiment 
with other methods of identifying employers. It is possible that asking graduates to 
supply employer names reduced the response rate and biased the study sample. There 
may be greater gain in increasing the response rate while sacrificing the chance to 
precisely pair employer and graduate surveys. 

In reviewing what we learned from this study we would say that, for our particular 
population, employers and graduates do not differ substantially in their assessments of 
the importance of skills for the workplace, or even in their assessments of particular 
graduates’ achievement of these skills. Although future survey designs may not be 
paired, we plan to continue to survey both these populations for several reasons. One 
is that we learned different information from the two groups on qualitative questions. 
Another is that proposed refinements in the response formats of quantitative questions 
may reveal differences, or even that differences may emerge as the industry and 
curricula change over time. And finally, although we feel that higher education should 
not be entirely directed by the needs of the workplace, neither do we feel that the needs 
of employers to have capable employees or of students to get satisfying jobs can be 
denied. Resnick (1987) so aptly pointed out the disparity between what is taught in 
school and the skills used in the workplace. Regularly collecting feedback from 
employers and graduates can help us improve curricula to close this gap. 
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Table I: 

Description of employer and graduate surveys 





Graduates Employers 


1) General 
Information 


6 questions collecting 
employment information 
containing (when inclusion 
criteria were met) a request for 
supervisor’s name and contact 
number. 


1 open-ended question 
requesting a description of the 
organization’s core activities. 


2) Important 
Skills for 
All Graduates 


22 closed response questions on 
a four-point scale indicating 
level of agreement, and one 
open-ended question inviting 
respondents to write in other 
essential skills that were not 
covered by the closed response 
items. 


Same as for graduates. 


3) Assessment of 
a Particular 
Graduate 


For each of the 22 areas 
identified in the previous 
section, closed response 
questions indicating a) current 
level of competency, and b) 
whether competency was 
developed during university 
training, on the same four-point 
scale. 


For each of the 22 areas 
identified in the previous 
section, closed response 
questions indicating the skill 
level of a particular graduate 
(named on the form) currently 
working for the employer, on 
the same four-point scale with a 
fifth option of “cannot 
comment.” 2 additional 
questions asking how many 
other graduates from the same 
university had worked with the 
employer in a given time 
period, and whether and how 
the identified graduate differs 
from other employees. 


4) Additional 
Open-ended 
Questions 


Three questions asking 
respondents to write in a) three 
strengths of the university 
program, b) three weaknesses, 
and c) any other comments. 


Two questions asking 
respondents to write in a) other 
areas of technical knowledge 
important for graduates to 
possess, and b) any other 
comments. 
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Table II: 

Comparison of Employers’ and Graduates’ Views on Important Skills 

(Paired) (p < .05) 





Skill 


I More 
| important j 
! to 

i Employers : 


[T^ore 1 

j important j No 
| to j Significant 

i Graduates j Difference 


1. 


Extensive practical knowledge 




! ✓ i 


2. 


Computer literate 


1 y \ 




3. 


Able to update professional knowledge 






4a. 


Possess effective: reading skills 




! ! y 


4b. 


listening skills 




\ \ y 


4c. 


oral communication 
skills 




y | 


4d. 


writing skills 




1 y 


5a. 


Able to communicate in: English 




i ✓ 


5b. 


Cantonese 




i y 


5c. 


Putonghua 




\ y 


6. 


j Gather and interpret information 


\ y 


7. 


| Define and solve problems i 


\ y 


8. 


| Adaptable and flexible to cope with 
| changing work environment 


\ y 


9. 


i Work autonomously I 


\ y 


10. | 


| Work co-operatively as a team member 


\ s | 




ii. i 


! Lead others effectively 




\ y \ 


12. I 


I Work effectively with multi-national 
| teams and projects 




1 ^ 


13. | 


! Be creative 




1 ✓ 


1 4 . 1 


Value and promote truth, accuracy, 
honesty, accountability and ethical 
| standards 


1 ✓ l 


| j 


15. | 


Accept responsibility 




I y 


16. i 


| Exercise professional judgment 




| 1 y 


17. ! 


| Recognize and respond to environmental 
i concerns 




\ y 
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Table III: 

Employers’ and Graduates’ Written Comments on Other Essential Skills 





Graduates (n=187) 


Employers (n=110) 


Number 
respondents/ 
number comments 


76/111 


51/59 


Most frequent 
comment category 


Graduates need general 
intellectual and analytical 
skills (ex. problem solving) 

(34 comments) 


Same as for graduates (with 
added emphasis that these 
skills would allow them to 
work more independently). 

(25 comments) 


Second most 
frequent comment 
category 


Graduates need stronger skills 
in particular technical areas 
(ex. law, building contracts) 

(15 comments) 


Same as for graduates. 
(14 comments) 


Third most 
frequent comment 
category 


Graduates need more practical 
or “hands-on” training. 

(6 comments) 


Same as for graduates. 
(6 comments) 
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Table IV: 

Comparison of Employers’ and Graduates’ Views on Achieved Skills 

(Paired) (p < .05) 





Skill 


nran i 

Higher 
According j 
to 

Employers j 


nrar 1 

i Higher 
| According j 
to 

| Graduates j 


No 

| Significant 
1 Difference 


1. 


Extensive practical knowledge 




y 




2. 


Computer literate 






1 L 


3. 


Able to update professional knowledge 






y 


4a. 


Possess effective: reading skills 




; : 


y 


4b. 


listening skills 


y 






4c. 


oral communication 
skills 






y 


4d. 


writing skills 






y 


5a. 


Able to communicate in: English 






y 


5b. 


Cantonese 






y 


5c. 


Putonghua 






y 


6. 


I Gather and interpret information 






y 


7. 


| Define and solve problems 




y j 




8- 


Adaptable and flexible to cope with 
| changing work environment 






y 


9. 


I Work autonomously 




y 




10. 


I Work co-operatively as a team member 


y 






11. 


| Lead others effectively 






y 


12. 


Work effectively with multi-national 
| teams and projects 






y 


13. 


| Be creative 


y 






14. 


Value and promote truth, accuracy, 
honesty, accountability and ethical 
| standards 






y 


15. 


| Accept responsibility 






y 


16. 


I Exercise professional judgment 






y 


17. 


Recognize and respond to environmental j 
| concerns 






y 
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Figure I: 

Quadrants of Effectiveness in Teaching Important Skills for Graduates 
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High Achievement 

Don 't bother ! 
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High Achievement 

Doing Well ! 


Unimportant Skills 
Low Achievement 

Who cares ! 


Important Skills 
Low Achievement 

Need to do better ! 
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Figure II: 

Scattered means of employers comments on importance and achievment 

of essential skills for graduates 
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